Planning for Distributed Earth Science Data Processing

An important challenge in Earth science processing is the large volume and distributed nature of the data required by many processing algorithms. Despite the increase in available bandwidth over the last several years, it is still often impractical, or at least very time-consuming, to acquire and locally stage the data prior to processing, because the volumes can run into tens or even hundreds of Gigabytes per day. In this paper, we describe a distributed system for Earth science data processing in which the control and execution of data-processing procedures are decoupled, allowing execution of independent data-processing components to be controlled remotely over the network. We use Web Services to export the interface of these components. Finally, we describe a planner-based agent that operates on the exposed Web Service components to automatically generate and execute data-flow programs to produce the requested products. We can then build entire processing pipelines that include pre-processing, processing and analysis of the results in an automated and efficient way, simplifying the development of distributed Earth science processing systems. We show an application of our system in the Terrestrial Observation and Prediction System (TOPS) whose goal is to provide a daily monitoring and prediction of numerous biospheric variables that are important indicators of the events happening within the Earth system. Finally, we show an application of our system to near-real-time fire monitoring and analysis.